Skip to content

x86: fix compressed displacements for scalar FP16 conversions - #336

Draft
agourakis82 wants to merge 1 commit into
netwide-assembler:masterfrom
agourakis82:fix/fp16-disp8-20261101
Draft

agourakis82 wants to merge 1 commit into
netwide-assembler:masterfrom
agourakis82:fix/fp16-disp8-20261101

Conversation

@agourakis82

@agourakis82 agourakis82 commented Nov 2, 2026 •

Copy link
Copy Markdown

Scalar FP16 conversions can encode a different memory address from the one written in the source. For example, vcvtsh2ss xmm0, xmm0, [rsi+4] currently emits a compressed displacement of 1, which the processor interprets as [rsi+2]. The same incorrect tuple metadata also makes ndisasm print the intended address instead of exposing the encoding error.

Use t1s16 for the ten FP16-source conversion forms: SH2SS, SH2SD, and the signed/unsigned, rounded/truncating integer conversions with 32- or 64-bit destinations. Their memory source is always 16 bits; EVEX.W selects the integer destination width and must not select a 4- or 8-byte displacement scale. Wider integer/FP sources retain their existing tuple types.

The new regression covers 110 instructions, including signed disp8 boundaries, noncompressible displacements, SIB addressing, register sources, implicit merge operands, masking, and unchanged m32/m64 controls. Refresh six existing 64-bit golden files; their lengths are unchanged and only ten displacement bytes change.

Validation on macOS ARM64:

  • Built nasm and ndisasm with Apple Clang 21.
  • Compared the new regression with LLVM MC 15.0.7 (x86_64, +avx512fp16): all 860 bytes agree. A separate C++23 checker extracted the reference bytes and compared files exactly; the unmodified baseline fails this check.
  • Replayed 60 existing positive and expected-negative cases from the affected conversion families and neighboring controls, using their declared NASM options and a C++23 byte comparator: all pass, including stderr references. This is a focused replay, not a claim that the complete Python-driven Travis harness was run.
  • Checked the original two conversions and VADDSH control with ndisasm and independently decoded the old bytes with LLVM MC.
  • git diff --check passes.

Related to #324. This addresses the scalar conversion cases, rather than claiming to fix every FP16 instruction discussed there. Native AVX-512 FP16 execution is additionally verified below; this does not claim coverage of every processor or every possible FP16 value.

Native cluster execution

Slurm job 21113 completed 0:0 in 63 seconds on cpuops-t560-proxmox, an Intel Xeon Gold 6526Y with AVX-512 FP16 and VL exposed to the process. Both assemblers were rebuilt on Linux from the original base fbdc88565 and this PR head 0cd2f34f1. A C++23 harness linked and executed their generated ELF instruction routines directly on the CPU, after checking CPU capability and explicitly setting round-to-nearest-even in MXCSR. No emulation was used.

The same 110 callable instruction cases ran 854 value/address assertions in each build. Neighboring half values were deliberately different from the value at the requested address. Cases include displacement boundaries, SIB/register sources, masking, merge operands, positive/negative values, signed zero, a subnormal, and rounded/truncating integer conversions.

baseline:  NATIVE_FUNCTIONS=110 NATIVE_TESTS=854 NATIVE_FAILURES=291 (exit 1)
candidate: NATIVE_FUNCTIONS=110 NATIVE_TESTS=854 NATIVE_FAILURES=0   (exit 0)
NATIVE_AVX512_FP16_ADDRESS_AND_VALUE_PASS

The 60 focused existing positive/negative cases were also replayed successfully in that Slurm job, including exact stderr comparisons. The candidate's 860-byte regression output still matches the independent LLVM MC reference byte for byte.

Use the fixed 16-bit scalar tuple for the FP16-source conversions. EVEX.W selects the integer destination width for SH2SI/SH2USI and must not change the memory displacement scale. The same metadata drives assembly and disassembly.

Cover all ten forms with signed displacement boundaries, SIB addressing, register operands and unchanged wider-source controls. Refresh the six existing 64-bit references whose compressed displacement bytes change. Addresses the scalar conversion cases reported in netwide-assembler#324.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant